The disclosed embodiments relate generally to methods and apparatus for monitoring data transmitted by streaming data sources, detecting data anomalies in the data, and transmitting notifications of such anomalies.
The amount of streaming data being generated is expanding exponentially from the growing variety of interconnected machines, devices, sensors, and consumer content. Moreover, the amount of data generated from any single source can be staggering.
Often, organizations implementing streaming data sources process the data transmitted by the streaming data sources via downstream processes that receive the data. Furthermore, data transmitted by a streaming data source may be obtained by multiple downstream processes. Some of these downstream processes may include batch processes. Since the input data for these batch processes is processed in batches or sets of records, the accuracy of the data processed by the batch processes is paramount.